[None][feat] Support DeepSeek-V4 model by lfr-0531 · Pull Request #14751 · NVIDIA/TensorRT-LLM

lfr-0531 · 2026-05-29T15:52:23Z

Description

This PR adds end-to-end support for the DeepSeek-V4 model on the PyTorch backend.
DeepSeek-V4 introduces sparse-MLA attention, and this change brings in the full path:
the compressor / indexer kernels and sparse KV-cache management, the QK-norm fused op,
MoE enablement (MEGAMOE DeepGEMM, EPLB), tokenizer chat-template handling, and the
tool / reasoning parsers. The change set is intentionally large; please refer to the
commit history for the per-feature breakdown.

Test Coverage

unittest/_torch/attention/sparse/deepseek_v4/* — sparse MLA, cache manager,
indices transform, o_proj, compressor kernel / module / tf32
unittest/_torch/modeling/test_modeling_deepseekv4.py
unittest/llmapi/test_deepseek_v4_tokenizer.py
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_2_model_mtp

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths.
If PR introduces API changes, an appropriate PR label is added.
Any new dependencies have been scanned for license and vulnerabilities.
CODEOWNERS updated if ownership changes.
Documentation updated as needed.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

lfr-0531 · 2026-05-29T16:19:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-29T16:25:42Z

PR_Github #51063 [ run ] triggered by Bot. Commit: aa1c243 Link to invocation

tensorrt-cicd · 2026-05-30T00:30:27Z

PR_Github #51063 [ run ] completed with state SUCCESS. Commit: aa1c243
/LLM/main/L0_MergeRequest_PR pipeline #40508 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-05-30T17:10:43Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-30T17:16:16Z

PR_Github #51198 [ run ] triggered by Bot. Commit: 94a2f25 Link to invocation

tensorrt-cicd · 2026-05-31T00:51:10Z

PR_Github #51198 [ run ] completed with state FAILURE. Commit: 94a2f25
/LLM/main/L0_MergeRequest_PR pipeline #40626 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-05-31T08:56:01Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-31T09:01:20Z

PR_Github #51248 [ run ] triggered by Bot. Commit: 94a2f25 Link to invocation

tensorrt-cicd · 2026-05-31T13:30:01Z

PR_Github #51248 [ run ] completed with state SUCCESS. Commit: 94a2f25
/LLM/main/L0_MergeRequest_PR pipeline #40670 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-05-31T15:05:02Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-31T15:10:31Z

PR_Github #51256 [ run ] triggered by Bot. Commit: 486b7d5 Link to invocation

tensorrt-cicd · 2026-05-31T23:09:19Z

PR_Github #51256 [ run ] completed with state FAILURE. Commit: 486b7d5
/LLM/main/L0_MergeRequest_PR pipeline #40677 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-06-01T00:38:40Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-01T00:44:07Z

PR_Github #51273 [ run ] triggered by Bot. Commit: 486b7d5 Link to invocation

tensorrt-cicd · 2026-06-01T06:55:33Z

PR_Github #51273 [ run ] completed with state SUCCESS. Commit: 486b7d5
/LLM/main/L0_MergeRequest_PR pipeline #40693 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-06-01T07:15:19Z

/bot run

tensorrt-cicd · 2026-06-01T07:20:59Z

PR_Github #51331 [ run ] triggered by Bot. Commit: 486b7d5 Link to invocation

tensorrt-cicd · 2026-06-01T11:43:31Z

PR_Github #51331 [ run ] completed with state FAILURE. Commit: 486b7d5
/LLM/main/L0_MergeRequest_PR pipeline #40741 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-06-01T11:57:22Z

/bot run

lfr-0531 · 2026-06-01T12:18:20Z

/bot run

lfr-0531 · 2026-06-01T15:22:35Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-01T15:28:32Z

PR_Github #51370 [ run ] triggered by Bot. Commit: 5de630c Link to invocation

tensorrt-cicd · 2026-06-01T21:33:03Z

PR_Github #51370 [ run ] completed with state FAILURE. Commit: 5de630c
/LLM/main/L0_MergeRequest_PR pipeline #40781 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

… RMSNorm fold-in (NVIDIA#13892) Signed-off-by: Mingyang Hao <mingyangh@nvidia.com> Signed-off-by: mingyangh <mingyangh@nvidia.com> Signed-off-by: Mingyang Hao <mingyangHao@users.noreply.github.com> Co-authored-by: Mingyang Hao <mingyangHao@users.noreply.github.com> (cherry picked from commit 50aa581) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com> Co-authored-by: Tracin <10434017+Tracin@users.noreply.github.com> Co-authored-by: Fanrong Li <lfr-0531@users.noreply.github.com> (cherry picked from commit 8eccf23) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com> Signed-off-by: Barry Kang <jinshik@nvidia.com> Co-authored-by: Barry Kang <jinshik@nvidia.com> (cherry picked from commit 13888d2) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…14129) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 7964767) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com> (cherry picked from commit 15ac239) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…DIA#14074) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com> (cherry picked from commit d178370) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com> (cherry picked from commit 5cf6e6a) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

NVIDIA#14124) Signed-off-by: Mingyang Hao <mingyangh@nvidia.com> (cherry picked from commit 82ebfc7) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Mingyang Hao <mingyangh@nvidia.com> (cherry picked from commit 4aad70b) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com> Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> Co-authored-by: Fanrong Li <lfr-0531@users.noreply.github.com> Co-authored-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit e6339b5) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…A#14000) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 12b8105) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Mike Iovine <miovine@nvidia.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> (cherry picked from commit 7fbe349) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Mingyang Hao <mingyangHao@users.noreply.github.com> Signed-off-by: Mingyang Hao <mingyangh@nvidia.com> Co-authored-by: Mingyang Hao <mingyangHao@users.noreply.github.com> (cherry picked from commit f833ad7) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Shicheng Li <shicli@nvidia.com> (cherry picked from commit d7d9036) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com> Co-authored-by: Fanrong Li <lfr-0531@users.noreply.github.com> (cherry picked from commit 5af1511) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 552de9e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…ymmBuffer (NVIDIA#14213) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com> (cherry picked from commit 5a72fb3) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…VIDIA#14219) Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> (cherry picked from commit bef645d) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…ttn (NVIDIA#14321) Signed-off-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> (cherry picked from commit 349f087) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…#14299) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com> Co-authored-by: Fanrong Li <lfr-0531@users.noreply.github.com> (cherry picked from commit 5d0a30e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Filtered out disaggregation and AutoDeploy-only changes for user/fanrongl/dsv4_model. Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com> (cherry picked from commit 325152c) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…eam` (NVIDIA#14245) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> Co-authored-by: peihengh <259410613+peihu-nv@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…4241) Signed-off-by: Lanyu Liao <lancelly@users.noreply.github.com> Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com> Co-authored-by: Lanyu Liao <lancelly@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…ety (NVIDIA#14297) Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

github-actions Bot assigned lfr-0531 May 29, 2026

lfr-0531 force-pushed the user/fanrongl/dsv4_model branch from e1c544f to aa1c243 Compare May 29, 2026 16:16

lfr-0531 added the api-compatible Accepted LLM API contract change that is backwards-compatible label May 29, 2026

lfr-0531 force-pushed the user/fanrongl/dsv4_model branch 2 times, most recently from 2608a8d to 94a2f25 Compare May 30, 2026 02:14

lfr-0531 force-pushed the user/fanrongl/dsv4_model branch from 94a2f25 to 486b7d5 Compare May 31, 2026 13:58

lfr-0531 force-pushed the user/fanrongl/dsv4_model branch from 486b7d5 to 5de630c Compare June 1, 2026 15:21

mingyangHao and others added 28 commits June 17, 2026 01:19

[None][feat] Enable MEGAMOE_DEEPGEMM backend for DeepSeek V4 (NVIDIA#…

b6484bf

…14129) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 7964767) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][perf] DSV4 multistream improvement for attention (NVIDIA#14142)

6b79771

Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com> (cherry picked from commit 15ac239) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][fix] Support DeepSeekV4 routing in perfect-router planner (NVI…

be77c7d

…DIA#14074) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com> (cherry picked from commit d178370) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[TRTLLM-12478][fix] Synchronize prepare inputs H2D copy (NVIDIA#14128)

2f76aa2

Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com> (cherry picked from commit 5cf6e6a) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][perf] DSV4 compressor: enable 4-warp Phase 3 reduction for MTP… (

22bd174

NVIDIA#14124) Signed-off-by: Mingyang Hao <mingyangh@nvidia.com> (cherry picked from commit 82ebfc7) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][fix] Make indexer top-K launch policy device-aware (NVIDIA#13886)

f402797

Signed-off-by: Mingyang Hao <mingyangh@nvidia.com> (cherry picked from commit 4aad70b) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[TRTLLM-12589][fix] Reset MoE A2A dispatch state on warmup OOM (NVIDI…

ad0f153

…A#14000) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 12b8105) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][feat] Enable 2 DSv4 perf optimizations by default (NVIDIA#14120)

6991c02

Signed-off-by: Shicheng Li <shicli@nvidia.com> (cherry picked from commit d7d9036) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][fix] Restore DSV4 routed-expert swiglu_limit on TRTLLM-Gen (NV…

d93363d

…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 552de9e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][fix] Avoid dp_size x ep_size double-count in MegaMoEDeepGemm S…

bfac884

…ymmBuffer (NVIDIA#14213) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com> (cherry picked from commit 5a72fb3) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][fix] DSv4: drop duplicate indexer_k_dtype kwarg on V4 sparse a…

ecb7fdb

…ttn (NVIDIA#14321) Signed-off-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> (cherry picked from commit 349f087) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[TRTLLM-12112][feat] Support v2 KV cache stats (NVIDIA#13953)

141a5e7

Filtered out disaggregation and AutoDeploy-only changes for user/fanrongl/dsv4_model. Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][perf] Enable PDL for DeepGEMM and related kernels (NVIDIA#14311)

5ddc357

Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com> (cherry picked from commit 325152c) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][fix] DSv4 indexer: stable radix aux scratch for CUDA Graph saf…

2f02674

…ety (NVIDIA#14297) Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][fix] Resolve DSv4 rebase fallout

3a5591f

Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][fix] Resolve DSv4 CI 43359 failures

d822d1d

Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

[None][fix] Stabilize MegaMoE MPI bootstrap

948ee55

Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

lfr-0531 force-pushed the user/fanrongl/dsv4_model branch from 9c86293 to 948ee55 Compare June 17, 2026 03:16

lfr-0531 closed this Jul 1, 2026

Uh oh!

Conversation

lfr-0531 commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

lfr-0531 commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 30, 2026

Uh oh!

lfr-0531 commented May 30, 2026

Uh oh!

tensorrt-cicd commented May 30, 2026

Uh oh!

tensorrt-cicd commented May 31, 2026

Uh oh!

lfr-0531 commented May 31, 2026

Uh oh!

tensorrt-cicd commented May 31, 2026

Uh oh!

tensorrt-cicd commented May 31, 2026

Uh oh!

lfr-0531 commented May 31, 2026

Uh oh!

tensorrt-cicd commented May 31, 2026

Uh oh!

tensorrt-cicd commented May 31, 2026

Uh oh!

lfr-0531 commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

lfr-0531 commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

lfr-0531 commented Jun 1, 2026

Uh oh!

lfr-0531 commented Jun 1, 2026

Uh oh!

lfr-0531 commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

lfr-0531 commented May 29, 2026 •

edited

Loading